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“If you torture the data long enough, it will 
confess to anything.” 
— Ronald Coase © 2020 Philip Koopman 4 
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Key Embedded Software Metrics Mell 


= Anti-Patterns: 
e Development effort > validation effort 
@ Too many lines of code per hour 
e Peer review finds <50% of all bugs 








= Healthy project metrics: 
e About 2-3 hours of validation effort per hour daglanihant 
— Tester:Developer head count ratio is about 1 to 1 
e Productivity of 1-2 lines of code per hour for solid software 
— This includes entire process (requirements through acceptance test) 
e Peer review should be finding >50% of all defects 
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Software = Design + Testing Ls Liggeerttty 


System level ‘Software level Validated software 
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IEC 2570/13 


EC 60730 Appliance Safety [IEC 60730] © 2020 Philip Koopman 3 
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Typical Effort Distribution aD 





= Tester to Developer ratio varies depending on situation 


1 tester 
1 tester 


e Web development: 
e Microsoft: 
e Aircraft controls: 


EMBEDDED SW PROJECT EFFORT 


25% 
DEV: Design & 
Implementation 


25% 
DEV: Peer Reviews 
& Unit Test 
9% SQA/PPQA 
45% 
TEST: Integration Test 
System Test 
Regression Tests 


per 5-10 developers 
per 1 developer 
~5 testers per 1 developer 






25%/75% Effort 


25% DEVELOPMENT 


DEV: Design & 
Implement 


DEV: Peer Review 
& Unit Test 





TEST: Integration Test 





& System Test 
50%/50% Head Count 
20 Person Project 73% VALIDATION 
& QUALITY 
10 DEVELOPERS 
50% 
gE 





9 TESTERS + 1 SQA/PPQA 
45% + 5% = 50% 
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Code Productivity 


= Productivity 1-2 lines of code/hr (including testers) 
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e Perhaps 3 lines/hr with Agile, but that speed increases quality risk 


= High lines of code/hr = cutting corners 
e Partial requirements, no design? 
e No peer reviews? 
e Only system level testing? 


= S25-S75 / line of source code 


e All-in cost, including entire V process, until field testing 
e “Maintenance’ can cost more, but might count as new project \ 





I WILL NOT CUT CORNERS! 
I WILL NOT CUT CORNERS! 


[Simpsons 7F11] 





© 2020 Philip Koopman 45 


Carnegie 


Peer Review Effectiveness aie 
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= Good peer reviews find 50%-70% of the defects 
e Fewer than 40%-50% of defects found in peer reviews mean they are BROKEN 





m Peer Reviews cost perhaps 5%-10% of total project cost 
e Let's do the math: 


— Peer reviews process about 100 lines of code per hour total 
— Three reviewers = 33 lines of code per person-hr 

= 0.033 hours per line of code reviewed (2 minutes) 
— 0.033 hours review / .5 hours per LOC total = 6.7% for code review 
— Plus review requirements & design ... but still a great ROI 


m Are peer reviews finding half your bugs? 
e Are you spreading them out or bunching them together? 
e If they're not finding bugs, consider improving review culture 
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Best Practices For Key Software Metrics Nilo 
= 2-3 hours of validation for each 1 hour of development 
e Head count ratio generally 1 Tester to 1 Developer 
e About 5% of effort for SQA 
= Code productivity of about 1 to 3 lines per hour 
e Ator above 3 lines/hr, you probably are cutting corners 
= Peer reviews should find 50% (or more) of defects 
e At about 5%-10% of total project cost 
= Metric Pitfalls 
e Use only metrics that provide value — dont go crazy with metrics! 
e Gaming the metric doesnt improve software quality 
e Reward/punish based on metric values will render metric useless 
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It would be a pure function if not for the side effects on your sanity 
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WHAT WERE THINGS | | TO COMPILE Your | | BEFORE GARBAGE COLLECTION 
LIKE IN THE OLD DAYS? CODE, YOU HAD TO epee 
T HEAR YOU HAD To... | | MAILIT 10 IGT. HE COMPUTER GOT FULL AND 
COMPILE THINGS FOR | | 10K U-6 WEEKS YOU HAD To THROW IT AWAY! 


1O'D ONLY FIT AS | preps sos PNOED HOLES 


( 





https://m.xked.com/1755/ 





© 2020 Philip Koopman 9 


